Technical Field
The present invention relates to machine learning and more particularly improvements to machine learning on networks.
Description of the Related Art
Machine learning methods are being applied across sensors, mobile and personal devices and even geographically distributed data centers. These devices generate and share the data and the computational costs of training a model. However, distributed learning in real world scenarios suffers from two issues. First, the various nodes in a real-world setting may suffer from intermittent network or node failures. For example, geographically separated data centers may suffer from communication delays or dropped packets. Second, the nodes in the distributed system such as the physical sensors may collect data points that are not randomly distributed across the nodes resulting in non-independent and identically distributed (non-i.i.d.) data across the nodes. Data-centers too, often collect non-random data, with each data center receiving data that is biased towards the geography where it is located. Often due to scale, privacy, or lack of a central coordinating resource, randomizing data may not always be possible. As a result, distributed training across these nodes in the presence of biased data at individual machines, based on simple techniques such as averaging of parameters may not work.